Allow tools whose schema is only known at runtime - #100
Conversation
7d3bc44 to
d88d9a0
Compare
|
Thanks for the contribution! The runtime tool schemas are lost when going through I think this path should pass the already-erased tools directly, for example: LocalLLMClient.llama(
url: url,
mmprojURL: mmprojURL,
parameter: parameter,
erasedTools: tools
)instead of: LocalLLMClient.llama(
url: url,
mmprojURL: mmprojURL,
parameter: parameter,
tools: tools.map { $0.underlyingTool }
)Also, I fixed the CI issue on the latest |
A host that gets its tools from a registry — an agent runtime, say — discovers them while running, as data: a name, a description and a JSON Schema string. There is no Swift type to hang `Arguments.argumentsSchema` on, so `AnyLLMTool.init(_:)` cannot express such a tool at all. Everything `AnyLLMTool` stores is already dynamic — `_argumentsSchema` is a dictionary, `_call` takes the raw arguments JSON — so this only adds the missing way in: an initializer taking those values directly, plus `DynamicLLMTool` to answer `underlyingTool` for a tool that has no type of its own. `LlamaClient` needed the same door. It took `[any LLMTool]` and erased them itself, which routes back through the static-schema initializer, so a caller holding an `AnyLLMTool` could not get it through. It now delegates to an `erasedTools:` variant that accepts tools already wrapped, and `LocalLLMClient.llama(…)` gained the matching overload. `LLMSession+Llama` used that same round trip: `makeClient` already receives `[AnyLLMTool]`, and mapping them back through `underlyingTool` re-erased them — emptying the schema of any tool built from runtime data. Both model factories now pass the erased tools straight through. Everything is additive: no existing call site changes.
d88d9a0 to
8174673
Compare
|
Good catch — you're right, and I'd missed it because my own usage calls Fixed as you suggested: both One thing worth flagging: |
The maintainer found a hole in tattn/LocalLLMClient#100 that our own use could not surface: `LLMSession+Llama` mapped its already-erased tools back through `underlyingTool` and re-erased them, which empties the schema of a tool built from runtime data. KaozKit calls `LlamaClient.responseStream` directly and never goes through `LLMSession`, so the local verification never touched that path. The fork now carries the fix and is rebased onto upstream main, which gained the CI stability work of #101 since.
tattn
left a comment
tattn
left a comment
There was a problem hiding this comment.
Thank you for taking care of this. The implementation looks good to me.
There are some interfaces that we may want to revise based on this change, but I think we can address those as needed and go ahead with the merge for now.
Thank you again.
2 participants
A host whose tools come from a registry discovers them while running, as data: a name, a description, and a JSON Schema string. There is no Swift type to hang
Arguments.argumentsSchemaon, soAnyLLMTool.init(_:)cannot express such a tool at all — which rules the library out for an agent runtime whose tool set is not known at compile time.Everything
AnyLLMToolstores is already dynamic:and
name/descriptionare already instance requirements onLLMTool. Only the way in is missing. This adds it:DynamicLLMToolcomes along to answerunderlyingTool, which is non-optional, for a tool that has no Swift type of its own.LlamaClientneeded the same door. It takes[any LLMTool]and erases them itself withtools.map { AnyLLMTool($0) }, which routes back through the static-schema initializer — so a caller holding an already-erased tool could not get it through. It now delegates to anerasedTools:variant, with a matchingLocalLLMClient.llama(…)overload.Everything here is additive: no existing call site changes, and the
[any LLMTool]initializer keeps its behaviour by delegating.Why
I am using
LocalLLMClientLlamaas a local-inference backend for a JS agent runtime, where an agent registers tools at run time and the host executes them itself — the model only needs to be told the schema and to emit the call. Verified end to end against Qwen2.5-3B: the schema reaches the chat template,LlamaToolCallParsersurfaces the call, and a multi-round exchange returns real tool output.LocalLLMClientLlamabuilds clean on this branch. Split from #99, which carries unrelated packaging fixes; the two are independent.